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Introduction 


ARPA's  current  biomedical  programs  are  generating  a  host  of  new  medical 
and  information  technologies  which  may  change  the  face  of  health  care 
(Satava,  1994).  Successful  deployment  of  these  innovations  will  depend  in 
large  part  on  the  clinical  acceptability  of  each  component,  as  well  as  their 
integration  into  the  fabric  of  everyday  clinical  practice. 

Clinical  acceptability  will  be  determined  by  at  least  four  factors: 

•  functionality  (Does  it  do  something  useful?) 

•  performance  (Is  is  effective /reliable/ safe  enough?) 

•  usability  (Can  the  average  medical  professional  use  it  easily?) 

•  cost  (Is  is  cost  effective  in  comparison  with  existing  alternatives?) 

One  common  theme  uniting  many  of  these  new  technologies  is  the  emerging 
electronic  patient  record.  It  is  clear  that  integration  must  take  place  at  the 
level  of  data  structures,  device  communications  and  database  access. 
Integrated  access  will  require  the  coordinated  interplay  of  a  variety  of  devices, 
including  traditional  (e.g,  workstations),  ubiquitous  (e.g.,  PDAs,  portable 
smart  tablets,  and  wall  boards),  and  personal  (e.g.,  body-worn,  such  as  HMDs). 
Equally  important,  however,  integration  must  take  place  at  the  level  of  the 
individual  user's  interface  to  the  data. 

Clinical  acceptability  may  best  be  served  by  starting  from  the  user's 
perspective  (unconstrained  by  technology)  and  letting  the  vision  of  the  ideal 
interface  drive  the  development  of  appropriate  technologies.  In  a  sense  the 
ideal  medical  interface  of  the  future  becomes  the  driving  problem  for  a  vast 
array  of  technology  innovations.  In  reality,  of  course,  both  technology-push 
and  applications-pull  must  work  in  tandem  to  move  the  interface  forward. 
In  our  project,  medical  interface  design  has  become  an  iterative  process  of 
unconstrained  imagination  followed  by  a  hard  engineering  analysis  of 
possible  implementation  approaches. 


Problem  -  Why  Look  at  the  Interface? 

The  need  for  a  ubiquitous,  intelligent  and  usable  human  interface  to 
physiological  monitoring  and  imaging  data  streams,  as  well  as  to  the  archival 
patient  record  and  reference  databases,  is  clear.  Our  search  for  an  integrated 
approach  is  motivated  by  a  number  of  observations: 

•too  much  clinical  data  to  assimilate 

•need  for  seamless  ubiquitous  access  ("anywhere,  anytime,  anyhow") 
•the  need  for  interoperability  among  devices  and  databases 
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•need  to  take  better  advantage  of  emerging  human  interface 
technology 

•need  to  support  evolution  of  medical  information  systems 
•need  to  tackle  clinical  acceptability  issues  up  front 

Advanced  Interface  Approaches 

Display  of  critical  medical  information  within  a  complex  clinical  setting 
presents  a  number  of  challenges,  aside  from  standardization  of  the  electronic 
patient  record.  Among  these  are  the  development  of  clinically  acceptable  data 
input  methods,  fail-safe  access  and  communication  methods,  and  the 
partitioning  of  the  user  interface  across  the  variety  of  potentially  useful 
devices. 

We  envision  a  rich  interface  "toolkit"  for  the  medical  practitioner  of  the 
future.  Along  with  traditional  workstations  and  device  monitors,  the 
medical  interface  environment  of  the  future  will  incorporate  four  key 
emerging  technologies: 

•  Virtual  Reality  (mediated  by  head-mounted  displays,  "cave"- 

type  environments,  and  "holographic"  or  stereographic 
systems) 

•  Augmented  Perception  (mediated  primarily  by  HMDs  and  other 

see-through  devices) 

•  Ubiquitous  Computing  (including  interface  devices,  such  as  wall 

panels,  wireless  pads  and  PDAs,  which  permit  ubiquitous 
access  to  a  common  database) 

The  medical  scenario  of  the  future  will  include  a  mix  of  these  interface 
approaches,  along  with  a  broad  array  of  input  modalities  and  methods, 
telepresence  technologies,  and  computer-mediated  assistance.  The  nature  of 
that  mix  will  be  highly  dependent  on  task,  specialty,  and  clinical  setting,  and 
presupposes  a  robust  and  reliable  electronic  patient  record.  To  be  clinically 
usable  these  systems  will  require  new  interface  metaphors  and  integration 
methods. 

Virtual  Retinal  Display  (VRD) 

Among  the  more  promising  new  enabling  technologies  for  advanced 
interfaces  is  the  Virtual  Retinal  Display  (VRD)  approach,  the  rapid 
modulation  and  scanning  of  light  directly  onto  the  retina  to  create  a  high- 
resolution  coherent  image.  The  fundamental  VRD  technology  and  a  variety 
of  VRD  devices  are  currently  under  development  at  the  HIT  Lab  (Kollin  and 
Tidwell,  1995;  Tidwell,  1995).  .  The  VRD  is  potentially  superior  to  all  other 
known  image  display  methods,  affording  maximum  brightness  and  a  spatial 
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resolution  limited  only  by  the  frequency  of  the  scanning  method,  rather  than 
by  the  image  source.  These  features  make  the  VRD  particularly  well-suited 
for  several  emerging  application  areas,  including  see-through  (augmented 
reality)  display,  small  light-weight  occluded  displays,  and  portable  high- 
resolution  projection  displays. 

The  mechanical  resonance  scanning  technology  developed  for  the  VRD  may 
also  prove  useful  for  spatial  data  acquisition  (position  tracking  and  scene 
acquisition).  These  technologies,  based  on  reflecting  scanned  light  out  into 
the  environment,  could  provide  the  low-latency  head  tracking  required  for 
many  augmented  reality  applications. 

Project  Objectives 

Clinical  acceptability  of  medical  information  systems  will  be  determined  in 
large  part  by  their  ease  of  use.  Interface  requirements  and  preferences  must  be 
determined  for  medical  practitioners  performing  various  medical  tasks  under 
a  variety  of  conditions.  HIT  Lab  researchers  are  collaborating  with  the 
University  of  Washington's  Image  Computing  Systems  Lab,  the  Biorobotics 
Lab  and  the  UW  School  of  Medicine's  Center  for  Videoendoscopic  Surgery, 
along  with  numerous  medical  advisers  from  a  wide  range  of  specialties,  to 
develop  novel  methods  for  interacting  with  clinical  data  and  emerging 
advanced  medical  technologies. 

In  particular,  we  are  developing  advanced  interface  technologies  which  will 
facilitate  time-critical  trauma  care  in  battlefield  situations  (with  clear 
applications  in  civilian  clinical  environments,  as  well).  The  primary  goal  of 
this  project  has  been  to  provide  some  of  the  basic  knowledge  and 
understanding  needed  to  develop  practical  human-computer  interfaces  for 
future  "integrated"  medical  information  systems.  The  specific  objectives 
addressed  by  this  project  were  to: 

•  Develop  a  set  of  design  guidelines  for  advanced  interfaces  that  integrate 

medical  concerns,  human  factors  requirements,  and  practical  system 
design. 

•  Establish  an  interface  testbed  to  evaluate  advanced  interface  concepts  and 

prototypes. 

•  Test  and  evaluate  advanced  interface  concepts. 

•  Develop  selected  prototypes  of  novel  data  interfaces. 

Methods 

This  phase  of  our  involvement  in  the  DARPA  Advanced  Biomedical 
Technology  Program  was  aimed  at  determining  the  interface  design 
guidelines  for  a  clinically  acceptable  integration  of  a  number  of  advanced 
medical  technologies,  being  developed  both  independently  and  under 


Human  Interface  Technology  Laboratory  -  3 


DARPA  sponsorship.  These  include  various  field  personnel  status 
monitoring  and  battlefield  telemedicine  technologies.  To  address  these 
complex  issues  we  adopted  an  approach  that  includes  a  cross-disciplinary 
team  of  physicians  and  engineers  working  jointly  to  develop  a  vision  of  an 
advanced  interface  to  medical  data  that  was  both  clinically  acceptable  and 
technologically  plausible. 


Design  Philosophy 

User  preferences  for  ideal  data  display  should  be  the  driving  force  behind 
interface  development.  The  precise  enabling  technologies  should  then  be 
determined  by  what  the  user  wants,  where  the  user  wants  it,  and  how  the 
user  wants  to  interact  with  it.  The  technological  solution  will  of  course  be 
subject  to  many  additional  constraints,  such  as  the  performance  characteristics 
and  costs  associated  with  each  approach,  ergonomics  and  clinical  usability, 
and  the  availability  of  reliable  power  and  communications  bandwidth. 

In  developing  these  concepts  the  project  team  has  drawn  heavily  from  the 
tradition  of  aircraft  cockpit  display  research  (Furness,  1969),  determining  for 
each  medical  specialty  a  set  of  ideal  data  display  configurations  for  various 
clinical  tasks.  The  basic  notion  is  to  design  integrated  interface  systems 
around  the  user's  needs  for  information,  relying  on  domain  experts  to  tell  us 
where,  when  and  how  information  should  be  displayed  from  their  particular 
perspective.  Customized  traversal  of  the  electronic  patient  record,  for 
example,  might  be  displayed  privately  in  head-stabilized  fashion,  while 
imaging  data  of  interest  to  several  parties  in  the  clinical  setting  might  be 
patient-stabilized  or  room-stabilized.  Given  user  requirements  and  other 
constraints,  implementation  options  can  then  be  designed  and  key  enabling 
technologies  developed. 

Storyboard  Approach 

Throughout  this  project  we  have  been  exploring  ideal  medical  information 
display  requirements  and  possible  interface  configurations  for  a  variety  of 
medical  specialties  performing  various  clinical  tasks.  As  a  starting  point,  each 
specialist  on  our  medical  advisory  team  was  asked  to  focus  on  their  role  in 
treating  a  hypothetical  case  set  ten  years  in  the  future.  The  case,  developed  by 
the  team,  was  designed  to  cover  many  of  the  advanced  biomedical 
technologies  being  sponsored  by  DARPA,  but  in  a  plausible  civilian  scenario: 

John  Doe  is  a  traveling  salesman  with  previously  undiagnosed  diabetes  and  a 
prior  history  of  heart  complaints.  His  home  wellness  system  has  warned  him 
that  his  diet  is  bad  and  he  needs  more  exercise,  but  he  has  not  complied.  Driving 
in  his  car  (equipped  with  a  GPS  beacon  in  case  of  accident)  and  wearing  his 
personal  status  monitor,  Mr.  Doe  experiences  severe  chest  pain  and  crashes  into  a 
tree.  Paramedics  are  dispatched  immediately  by  the  regional  medical  C3  center 
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and  arrive  to  find  his  knee  crushed  and  his  leg  apparently  ischemic.  A 
cardiologist  is  consulted  while  en  route  to  the  appropriate  trauma  center.  It  is 
determined  that  Mr.  Doe  is  experiencing  an  evolving  myocardial  infarction, 
which  must  be  stabilized  prior  to  surgery  to  repair  his  damaged  popliteal  artery. 
Following  surgery,  Mr.  Doe  is  placed  in  an  instrumented  recovery  bed  which 
functions  both  as  a  portable  ICU  and  as  a  passive  rehabilitation  environment. 

Despite  an  acute  cardiac  incident  during  recovery,  Mr.  Doe  enjoys  a  successful 
rehabilitation,  and  is  discharged  to  the  care  of  a  home  health  nurse  and  his 
reprogrammed  set-top  wellness  advisor. 

As  a  template  for  each  medical  specialist's  interface  concepts  we  have  drawn 
heavily  from  the  pioneering  work  on  advanced  cockpit  displays  introduced  by 
the  Air  Force's  SuperCockpit  program  (Furness,  1986,  1988).  These 
revolutionary  interface  concepts  include: 

•  circumambient  spatial  display  (i.e.,  immersive  information 

environment) 

•  augmented  reality  (i.e.,  computed  generated  graphics  and  video  overlaid 

on  the  natural  scene) 

•  data  objects,  rather  than  windows,  are  the  unit  of  display 

•  multiple  imagery  stabilization  points  (as  illustrated  in  Figure  1) 

•  multi-sensory  multi-modal  interaction  with  data  objects 


EARTH 
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Figure  2  illustrates  one  example  of  such  an  interface  design  from  this 
emergency  room  scenario.  In  this  case  the  viewpoint  is  that  of  a  cardiologist 
who  has  been  called  to  the  trauma  center.  In  addition  to  rapidly  traversing 
the  patient's  archival  record  in  his  upper  field  of  view,  the  cardiologist  is 
monitoring  the  patient's  vital  signs  and  EKG,  while  consulting  with  the 
trauma  doc  and  the  telepresent  vascular  surgeon  about  the  arteriogram  and 
the  stability  of  the  patient's  cardiac  function. 
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FIGURE  2.  Physician-designed  visual  display  for  cardiologist  in  trauma  center. 


The  12-lead  EKG  or  the  patient's  vital  signs,  appear  stabilized  with  respect  to 
the  room  or  the  patient  ("world-stabilized"  or  "patient-stabilized").  Since  the 
EKG  is  of  interest  to  most  people  in  the  room,  a  likely  display  option  for  that 
data  element  is  a  live  wall  panel.  In  contrast,  data  of  interest  to  only  one  or  a 
few  individuals  are  presented  via  a  personal  (rather  than  public)  display 
device.  An  archival  timeline,  for  example,  which  represents  a  customized 
exploration  of  the  patient  record,  is  presented  here  in  the  upper  portion  of  the 
cardiologist's  field  of  view  in  "head-stabilized"  fashion.  The  presentation  of 
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various  data  fields  in  the  patient  record,  such  as  patient  name  or  referring 
physician,  could  be  color  coded  by  the  user.  Patient  record  navigation  could 
be  controlled  by  a  variety  of  modalities,  including  voice,  eye  movement,  or 
gesture.  Critical  points  along  the  timeline  could  thus  be  exploded  to  reveal 
graphical  representations  of  study  results  or  clinical  notes. 

Data  which  the  cardiologist  wants  to  keep  accessible  but  not  always  in  his  or 
her  field  of  view  might  be  "body-"  or  "vehicle-stabilized",  virtually  traveling 
with  the  cardiologist  as  he  moves  around  the  facility  but  not  continually  in 
the  visual  field.  Personal  notes  are  a  good  candidate  for  such  a  display. 
Finally,  in  this  example,  we  show  that  the  telepresent  consultant  should  be 
placed  (virtually)  in  a  location  that  allows  for  seemingly  natural  interaction 
about  the  subject  of  mutual  interest  (in  this  case  an  imaging  study  of  the 
damaged  artery,  overlaid  on  the  patient's  knee). 


LIMIT  Testbed 

In  fleshing  out  the  storyboard  scenario  and  discussing  various  design  options, 
it  became  apparent  that  we  needed  to  address  many  of  these  questions  within 
the  context  of  a  more  realistic  clinical  environment.  To  accomplish  this  we 
constructed  a  virtual  reality  simulator  for  designing  and  assessing  new  ways 
of  interacting  with  clinical  data.  This  VR  testbed,  called  the  LIMIT  (or 
"Laboratory  for  Integrated  Medical  Interface  Technology"),  consists  of  several 
photorealistic  texture-mapped  environments  constructed  from  video  image 
scans  of  clinical  settings,  rudimentary  male  and  female  patient  models,  and  a 
collection  of  clinical  data  objects  which  can  be  manipulated,  scaled  and 
positioned  anywhere  within  the  environment  (in  anticipation  of  clinically 
acceptable  augmented  reality  and  ubiquitous  computing  interfaces).  Two 
physicians  manipulating  data  objects  in  the  LIMIT'S  Virtual  ER  can  be  seen  in 
Figure  3. 


FIGURE  3.  Configuring  the  data  display  environment  in  the  LIMIT  virtual  ER. 
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Building  upon  concepts  developed  in  our  earlier  discussions,  the  primary 
purpose  of  the  LIMIT  is  to  provide  a  testbed  for  designing  the  clinical  interface 
of  the  future.  We  anticipate  that  other  uses  will  include: 

•  clinical  protocol  development 

•  individual  and  team  training,  and 

•  determining  performance  requirements  for  electronic  patient  data 

delivery  systems 

One  of  the  primary  advantages  of  this  approach  to  interface  design  is  that  VR 
affords  the  possibility  of  performing  pseudo-field  experiments  in  which 
extraneous  variables  are  controlled  for  and  variables  of  interest  are 
manipulated  in  ways  that  approximate  real  clinical  environments.  Since 
interaction  with  the  virtual  world  is  mediated  by  the  computer,  it  can  be 
thought  of  as  an  instrumented  research  environment. 

Current  Capabilities 

The  LIMIT  is  currently  implemented  on  two  VR  software  platforms,  both 
running  on  SGI  reality  engine  machines:  Division's  DVS,  and  Sense8's 
WorldToolkit.  Both  versions  make  use  of  code  enhancements  developed  in- 
house  and  include  support  for  multi-modal  (voice  and  gesture)  interaction. 
Three  immersive  texture-mapped  environments  are  currently  available: 

•  an  emergency  room  from  Harborview  Medical  Center,  Seattle  (a 

Level  1  trauma  center) 

•  an  operating  room  from  the  18th  MASH  unit  (as  configured  at 

Madigan  Army  Hospital,  Ft.  Lewis,  Spring  1996) 

•  a  treatment  room  from  the  18th  MASH  unit  (as  configured  at 

Madigan  Army  Hospital,  Ft.  Lewis,  Spring  1996) 

The  LIMIT  development  team,  which  consists  of  both  HIT  Lab  software 
engineers  and  our  consulting  health  care  professionals,  has  focused  the 
majority  of  its  efforts  thus  far  on  emergency  room  procedures.  However,  the 
data  display  methods  and  interaction  tools  we  have  developed  are  designed 
for  use  in  any  clinical  environment. 

Data  objects  are  rendered  independently  and  have  several  generic  capabilities: 

•  multiple  frames  of  reference  for  location  stabilization,  enabling  head- 

stabilized,  patient-stabilized,  and  room-stabilized  data 

•  object  orientation  adapts  to  the  physician's  position  by  bill-boarding 

•  all  objects  can  be  arbitrarily  scaled  and  positioned 
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•  all  objects  can  be  grabbed  and  relocated  by  direct  manipulation  (using 

both  linear  and  non-linear  mapping  of  the  user's  physical  and 
virtual  hand) 

•  object  visibility  can  be  toggled  by  keyboard  or  voice  command 

Data  objects  are  currently  encoded  as  either  texture  maps  or  files  of  integer 
values.  Texture  maps  have  been  acquired  for  rhythm  strips,  intake  charts, 
patient  demographics,  teleconsultant  video  frames,  and  various  radiology 
images.  Real-time  data  streams  have  been  implemented  for  trend  data,  and 
navigation  methods  are  being  implemented  for  multi-image  radiology 
studies  using  the  HIT  Lab's  "working  surfaces"  interface  approach  (Hollander 
and  Tidwell,  1997).  As  an  aid  to  run-time  reconfiguration  of  the  interface  by 
immersed  physicians,  we  have  also  added  a  non-linear  mapping  of  the  user's 
reach  for  direct  selection  and  manipulation  of  both  close  and  remote  objects 
(Poupyrev  et  ah,  1996). 

In  addition  to  these  generic  data  "presentation"  techniques,  we  are  exploring  a 
number  of  data  "representation"  techniques  which  make  use  of  3D 
interaction  capabilities.  Several  novel  data  representation  methods  have 
been  implemented  for  ECG  data  (as  described  in  Kaufman  et  ah,  1997)  and 
others  are  being  designed  for  lab  values  and  vital  signs  data. 

Three  testbed  enhancements  are  currently  under  development: 

•  secondary  task  loading  and  other  measures  for  assessing  display 

effectiveness 

•  multiple  participants,  with  shared  and  independent  data  objects 

•  porting  from  VR  to  Augmented  Reality  test  environment,  using  our 

"shared  space"  collaborative  AR  software  for  mutual  registration 
and  calibration 

The  secondary  task  we  are  implementing  is  a  waveform  monitoring  task,  in 
which  the  user  must  watch  to  see  when  a  trend  drifts  outside  a  threshold 
window  and  then  indicate  the  appropriate  baseline  adjustment.  This  task  is 
performed  while  also  monitoring  the  data  object  of  interest  (the  primary 
task).  In  the  tradition  of  avionics  display  research,  performance  on  the 
secondary  task  is  taken  as  an  additional  measure  of  the  goodness  of  the 
primary  data  display. 

Adding  support  for  multiple  participants  will  enable  us  to  explore  a  host  of 
issues  related  to  inter-user  negotiation  of  common  display  space,  methods  for 
sharing  data  and  communicating  about  common  artifacts,  and  team 
coordination  and  training  issues.  And  porting  to  a  shared  AR  environment 
will  enable  us  to  test  new  prototype  displays  and  interaction  techniques  in 
actual  clinical  settings. 
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Results 


Medical  Advisory  Team 

The  following  individuals  have  been  active  as  HIT  Lab  medical  advisers 
during  Phase  1  of  this  project,  and  will  continue  to  be  available  as  consultants 


in  follow-on  phases: 

Anesthesiology/Patient  Monitoring 

Peter  Dunbar,  MD 
University  of  Washington 
Harborview  Medical  Center 
pjdunbar@u.washington.edu 

Biological  Structures/Digital  Anatomy 

Jim  Brinkley,  MD/PhD 
University  of  Washington 
brinkley@u.  washington.edu 

Cardiology/Medical  Informatics 

Stan  Kaufman,  MD/PhD 
University  of  Washington 
Seattle  VA  Medical  Center 
tinman@u.  washington.edu 

Dermatological  Surgery 

Peter  Odland,  MD 
University  of  Washington 
odland@u.  washington.edu 

Nursing/Electronic  Patient  Record 

Kerry  Meyer,  PhD 
University  of  Washington 
Seattle  VA  Medical  Center 
petrucci@u.  washington.edu 

Otolaryngol  ogy 

Maj.  Chuck  Edmond,  MD 
Madigan  Army  Medical  Center 
cedmond@hitl.washington.edu 

Vestibular  Neurotology 

Erik  Viirre,  MD/PhD 
University  of  Washington /HIT  Lab 
Virtual  i-O,  Inc. 
erikv@hitl.washington.edu 


Radiology 

David  Haynor,  MD/PhD 
University  of  Washington 
Seattle  VA  Medical  Center 
haynor@u.washington.edu 

Steven  Carter,  MD 
University  of  Washington 
ACTS  Teleradiology  Project 
carter@u.  washington.edu 

Radiation  Oncology 

Case  Ketting,  PhD 

University  of  Washington 

case@violinl.radonc.washington.edu 

Ira  Kalat,  PhD 

University  of  Washington 

ira@violinl.radonc.washington.edu 

Surgery 

Mika  Sinanan,  MD 
Center  for  Videoendoscopic  Surgery 
University  of  Washington 
mssurg@u.  washington.edu 

Carlos  Pellegrini,  MD 
Chief  of  Surgery 
University  of  Washington 

Alex  Porter,  MD 
University  of  Washington 
porteral@u.washington.edu 

Trauma/Emergency  Medicine 

Joe  Gifford,  MD 
Northwest  Hospital 
jgiff@halcyon.com 
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The  backbone  of  our  project  has  been  the  involvement  of  a  strong  team  of 
medical  advisors,  including  specialists  in  anatomy,  surgery,  radiology, 
otolaryngology,  ophthalmology,  cardiology,  orthopaedics,  emergency 
medicine,  anesthesiology,  family  medicine,  radiation  oncology,  nursing, 
medical  informatics,  and  electronic  patient  record  systems.  Our  medical 
advisers  meet  weekly  with  HIT  Lab  engineers  and  scientists  to  exchange 
information  about  emerging  technologies  and  clinical  requirements,  and 
individual  advisers  have  worked  closely  with  lab  engineers  to  specify  and 
develop  clinically  appropriate  prototype  displays  and  interface  methods. 

Other  physicians,  nurses  and  allied  medical  personnel  from  a  variety  of 
medical  specialties  have  occasional  involvement  with  the  project,  and  are 
available  for  advice  and  consultation.  Our  DARPA-sponsored  activities  have 
provided  a  sounding  board  for  new  medical  interface  ideas  and  have 
catalyzed  a  number  of  other  collaborative  projects.  Perhaps  the  greatest 
strength  of  this  working  group  is  that  it  provides  an  opportunity  to  explore 
both  specialty-specific  and  clinically  generic  interface  needs. 


Review  of  Emerging  Technologies 

Anticipating  the  interface  of  the  future  requires  that  we  be  on  top  of  the  latest 
developments  in  computing,  communications,  and  interface  technologies. 
Through  its  ongoing  Knowledge  Base  program  efforts  the  HIT  Lab  has 
become  a  nexus  of  on-line  information  about  emerging  technologies  (via  the 
Virtual  Worlds  Consortium,  moderation  of  the  internet  newsgroup 
sci. virtual-worlds,  our  active  WWW  and  FTP  sites,  and  through  production 
and  maintenance  of  numerous  on-line  resource  guides),  and  we  have  worked 
closely  with  our  research  librarians  and  our  engineering  staff  to  remain 
current  in  the  area  of  advanced  interface  technologies  in  medicine.  A 
comprehensive  guide  to  relevant  web  sites,  vendors  and  published  literature 
is  located  at  www.hitl.washington.edu/projects/knowledge_base/medvr/. 


Concept  Demonstrations 


In  addition  to  assessing  the  medical  interface  advances  of  other  groups, 
collaborating  labs  working  on  this  project  have  developed  demonstrations  of 
several  enabling  technologies  and  promising  approaches,  including: 

•  teleradiology  consultation  tools  and  optimized  user  interface  for 

telemedicine  (using  the  Mediastation  5000  workstation) 

•  fractal  tissue  generation  (for  rapid  model  building  when  patient- 

specific  data  is  not  required) 

•  optimizing  photorealism  in  tissue  modeling  using  texture  mapping 

methods 
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•  multi-modal  expert  surgical  assistance  (using  voice,  position,  and 

instrument  location)  for  simulation  and  real-time  procedure 
support 

•  collaborative  (multi-participant)  augmented  reality 


Interface  Design  Methodology 

As  a  vehicle  for  organizing  the  vast  amounts  of  information  and  issues 
relevant  to  advanced  interfaces  for  medicine  we  have  adopted  a  scenario- 
based  approach  to  interface  design  (as  described  above).  Working  through  a 
hypothetical  medical  case  set  in  the  year  2005,  our  medical  advisers  have 
formulated  descriptions  of  anticipated  medical  advances  and  an  ideal 
integrated  interface,  as  they  carry  out  various  tasks  related  to  the  case.  In 
group  brainstorming  sessions  we  are  able  to  develop  candidate  concepts  in 
medical  interface  design  and  to  anticipate  issues  that  will  arise  in  multi¬ 
participant  situations. 

In  developing  new  medical  interface  concepts  we  have  drawn  heavily  from 
previous  DOD  work,  in  particular  the  SuperCockpit  program  sponsored  by 
the  U.S.  Air  Force  throughout  the  1980s.  Among  the  most  fruitful  of 
these  is  the  framework  the  SuperCockpit  program  provided  for  visual 
imagery  stabilization  based  on  a  set  of  distinct  reference  points  (the  user's  eye, 
head  or  vehicle,  along  with  the  earth  and  inertial  space).  We  have  adapted 
this  framework  to  applications  in  the  medical  domain,  as  described  elsewhere 
(Weghorst,  1996b,  1997). 

A  second  major  influence  on  our  design  approach  has  been  the  tradition  of 
multi-modal  interaction  with  computing  systems.  In  particular,  we  are 
leveraging  off  of  other  HIT  Lab  projects  which  combine  gesture,  voice,  and 
gaze  behavior  to  enable  a  more  intuitive  and  natural  interface  (Billinghurst  et 
ah,  1996). 

VR  Clinical  Simulation  Testbed 

We  have  developed  a  fairly  extensive  virtual  reality  simulation  of  a  possible 
medical  interface  environment  of  the  future.  In  addition  to  the  basic  LIMIT 
clinical  settings  and  data  interaction  methods,  we  have  accomplished  the 
following: 

•  initial  characterization  of  data  types  and  display  requirements  for  a 

wide  variety  of  clinical  data  objects,  including  vital  signs  data, 
clinical  images,  lab  results,  and  anesthesiology  data 

•  specification  (by  clinical  specialists)  of  desired  interaction  methods  for 

each  of  the  above  data  types 
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•  development  and  demonstration  of  novel  presentations  of  ECG  (and 

other  trend  data),  including  real-time  waveform  production  and 
an  abstract  heart  model  which  shows  correlated  real-time 
electrical  activity 

•  development  of  display  performance  metrics  and  assessment 

methods 

Surgical  Training  Initiative 

As  an  offshoot  of  DARPA  project  discussions,  the  UW  Surgery  Department's 
Center  for  Videoendoscopic  Surgery  (CVES)  has  incorporated  a  simulation 
thrust  into  their  surgical  resident  training  program.  Current  activities 
include  analysis  of  the  costs  and  benefits  of  physical  and  virtual  simulation 
(as  well  as  hybrid  methods)  for  various  levels  of  training  (e.g.,  skills  training 
vs.  procedure  training).  Physical  simulation  "packs"  for  basic  skills  and  for 
laparoscopic  cholecystectomy  have  been  developed  by  a  regional  company, 
SimuLabs,  and  incorporated  into  the  surgery  curriculum.  The  CVES  will  also 
serve  as  a  test  and  development  site  for  HighTechsplanations'  TELEOS 
simulation  software.  A  joint  study  demonstrating  a  positive  correlation 
between  videogame  expertise  (two-handed  3D  Tetris)  and  laparoscopic  skill 
acquisition  in  first-year  surgery  residents  was  reported  at  the  Medicine  Meets 
Virtual  Reality  meeting  in  January  1996  (Weghorst  et  al,  ,1996a). 


Spin-off  Collaborations 

Our  DARPA-sponsored  activities  have  also  significantly  strengthened  several 
other  new  joint  initiatives,  including  HIT  Lab  work  on  surgical  simulator 
fidelity  requirements  analysis  and  evaluation  for  the  Army-sponsored  ENT 
simulator  effort  with  Lockheed  Martin  and  Madigan  Army  Medical  Center; 
joint  proposals  with  the  telemedicine  evaluation  team  for  Project  Seahawk; 
ongoing  discussions  of  advanced  displays  with  several  medical  system 
vendors  and  Harborview  Medical  Center;  interface  evaluation  for  ACTS 
satellite-based  teleradiology  with  the  Department  of  Radiology;  and  a  new 
technology  development  project  focused  on  "shared  space"  (collaborative 
augmented  reality)  jointly  sponsored  by  Virtual  i-O  and  the  Washington 
Technology  Center. 

Although  official  commercial  partnerships  have  not  yet  been  established, 
discussions  are  under  way  with  PhysioControl  (heart  monitors  and 
defibrillators),  EmTek  (electronic  patient  data  systems),  ATL  (ultrasound 
equipment),  and  others,  to  incorporate  live  data  streams  into  the  LIMIT  and 
to  provide  a  prototyping  testbed  for  producers  of  clinical  equipment. 


Human  Interface  Technology  Laboratory  -  13 


Discussion 


Of  the  project  outcomes  detailed  above  we  believe  the  most  significant  is  the 
evolution  of  the  clinical  simulation  testbed.  By  providing  a  sense  of  presence 
in  a  controlled  clinical  setting,  we  are  able  to  prototype  and  test  candidate 
interface  elements  in  an  environment  which  approximates  the  target 
application  setting. 

The  LIMIT  testbed  provides  unique  support  for: 

•  exploration  and  rapid  prototyping  of  clinical  data  display  methods  and 

interface  metaphors,  including  traditional  WIMP  workstations,  ubiquitous 
computing  interfaces,  novel  teleconferencing  methods,  and  both  virtual 
(VR)  and  augmented  reality  (AR)  methods. 

•  clinical  display  performance  metrics,  including  many  adapted  from  aircraft 

displays  research,  for  assessing  both  generic  and  task-specific  display 
effectiveness 

•  clinical  protocol  development  and  training  within  this  set  of  simulated 

environments 

•  methods  for  rapid  construction  of  additional  texture-mapped  environments 

and  data  objects 

The  LIMIT  can  be  readily  adapted  to  support  a  variety  of  image,  trend,  and 
alphanumeric  data.  In  addition  to  vital  signs  data,  lab  results,  and  radiology 
and  pathology  images,  the  LIMIT  can  be  further  adapted  to  support  research 
on  candidate  interfaces  for  teleconsultation  and  teleoperations.  By 
incorporating  alpha-mapped  live  video  textures,  for  instance,  we  will  be  able 
to  experiment  with  various  representations  of  remote  consultants.  And  by 
simulating  telemedicine  activities  within  a  distributed  virtual  environment 
we  will  be  able  to  develop  and  assess  various  methods  of  communication 
with  the  local  participants  and  with  shared  artifacts. 

Our  discussions,  story-boarding,  and  virtual  simulation  methods  have  led  us 
to  the  following  general  conclusions: 

•  The  interface  of  the  future  must  advance  beyond  "what-you-see- 

is-what-you-get"  to  "where-you-need-it-is-where-you-get-it". 

•  Spatial  interface  methods  demand  interface  metaphors  that  go 

beyond  "desktop  windows"  to  make  optimal  use  of  human 
3D  capabilities. 

•  The  functional  integration  of  "ubiquitous",  immersive,  and 

traditional  displays  requires  a  seamless  and  intuitive  user 
interface  to  the  electronic  patient  record. 
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•  Interface  demands  differ  across  specialities,  tasks,  clinical  settings, 

and  individuals,  and  should  therefore  remain  fluid,  while 
supporting  intelligent  defaults  and  individual  preference 
configurations. 

•  The  medical  interface  of  the  future  must  take  advantage  of  both 

artificial  intelligence  (AI)  and  intelligence  augmentation  (IA) 
technologies,  in  combination  with  multi-modal  interface 
methods. 

Although  individual  clinicians  performing  different  tasks  prefer  different 
display  configurations,  certain  conventions  are  beginning  to  emerge.  Visual 
clutter  in  the  central  field-of-view,  for  instance,  will  not  be  acceptable  when  a 
clear  view  of  the  patient  is  required  (as  it  is  for  the  trauma  doc,  or  in  open 
surgery).  Critical  data  should  be  more  readily  accessible  (perhaps  head- 
stabilized),  and  transient  data  should  be  dismissed  after  it  is  noted. 

Future  Work 

Ultimately  interface  configuration,  training  and  operational  systems  will  be 
mediated  by  an  integrated  software  infrastructure;  the  design  of  such  a  system 
will  benefit  greatly  from  both  computational  and  interface  simulation. 
Building  upon  our  experience  in  Phase  1,  our  primary  objective  for  Phase  2  of 
this  project  is  to  construct  an  enhanced  virtual/augmented  reality 
environment  which  will  allow  us  to  rapidly  simulate  various  clinical 
information  human  interface  options.  Such  an  environment  will  permit 
interface  design  both  unconstrained  by  existing  technologies  (i.e.,  the  ideal 
clinical  info  display)  and  then  constrained  in  various  ways  (e.g.,  limited  to 
ubiquitous  devices,  or  limited  by  noise  or  communications  lag). 

As  with  the  current  LIMIT  environment,  research  participants  will  be 
medical  professionals  who  will  be  given  tools  to  rapidly  build  and  test  their 
own  information  environments,  working  alone  and  on  collaborative  clinical 
tasks. 

Phase  2  Objectives 

Specific  objectives  of  Phase  2  are  as  follows: 

Augmented  reality  emulation.  As  suitable  see-through  visual  displays 
become  available  during  Phase  2  we  will  begin  to  prototype  clinical  AR 
interface  emulations.  Augmented  reality  simulations  will  display  data  objects 
within  the  physician's  local  workspace,  either  unconstrained  spatially  or 
overlaid  on  tracked  physical  devices. 
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Real-time  data  acquisition.  Enhance  simulator  realism  by  incorporating 
realistic  patient  data  streams.  Discussions  have  begun  with  Emtek,  Quantum- 
Siemens,  SpaceLabs,  and  other  imaging  and  patient  monitoring  companies  to 
simulate  advanced  interfaces  to  their  systems. 

Configuration  tools.  Develop  additional  tools  for  data  display  configuration 
by  clinicians.  These  would  include  immersive  methods  for  defining  defaults, 
more  fluid  generic  interaction  methods,  and  additional  data-specific 
presentation  methods. 

System  performance  studies.  Study  clinical  acceptibility  under  various 
simulated  performance  constraints  (including  communications  lag  and 
reliability,  display  device  characteristics,  and  database  access  errors). 

Clinical  teams.  Simulate  multi-participant  scenarios  (data  display  collision, 
etc).  Incorporate  multi-participant  software  tools  as  distributed  VR  systems 
(such  as  the  HIT  Lab's  Greenspace  project)  evolve. 

Electronic  patient  record.  Incorporate  expert  system  assistance,  appropriate 
filtering  tools  for  the  emerging  multi-media  electronic  patient  record,  and 
other  AI  archive  navigation  tools. 

Sytem  performance  requirements.  Develop  performance  specifications  and 
high-level  software  architecture  for  Phase  3  prototype  development  effort. 

For  this  enhanced  simulation  phase  we  will  incorporate  as  many  simulated 
target  technologies  as  possible.  This  will  allow  us  to  rapidly  assess  the  clinical 
utility  of  various  configurations  of  physical  devices,  and  to  anticipate  how 
they  might  be  used,  prior  to  committing  to  an  implementation  system 
architecture.  System  parameters  to  be  simulated  include: 

•  data  display  constraints  (e.g.,  spatial  resolution  requirements) 

•  target  interface  device  constraints  (e.g.,  ubiquitous  only,  or 

restricted  field  of  view) 

•  data  communications  lag 

•  lag  variability 

•  data  integrity  and  noise  requirements 

•  effects  of  bandwidth  constraints 

•  resource  management  strategies 

•  resolution  strategies  for  multi-participant  display  conflict 

•  responses  to  system  failures 

Phase  3  Objectives 

In  Phase  3  we  propose  to  develop  an  actual  working  prototype  of  the  required 
interface  integration  software.  The  interface  hardware  implementation  will 
be  open  and  user-definable.  For  example,  the  display  could  be  private  (HMD) 
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or  public  (terminals,  electronic  whiteboards).  In  the  case  of  public  displays, 
protocols  for  who  controls  the  contents  of  that  display  will  have  to  be 
established;  in  the  case  of  private  displays,  they  will  be  tailored  to,  and 
customized  by,  the  individual  users.  Similarly,  interaction  with  the  system 
could  be  through  mice,  gestural  and  gaze  recognition,  voice  commands,  or 
other  methods. 

Specific  objectives  of  Phase  3  are  as  follows: 

Refine  high-level  software  architecture.  Building  upon  results  of  Phase  2, 
define  (software)  object  classes  and  hierarchy,  specify  methods  and  object 
behaviors. 

Construct  prototype  system.  Build  a  software  environment  for  intuitive 
integration  of  various  clinical  data  streams,  including  display  of  personnel 
status  monitor  (PSM)  data,  telemedicine,  and  electronic  patient  record  data, 
using  ubiquitous,  traditional,  and  personal  interface  devices.  VRD-based 
devices  and  other  high-resolution  high-luminance  displays  will  be 
incorporated  as  they  become  available. 

Clinical  field  demonstration.  Study  clinical  acceptibility  of  the  prototyped 
system  in  an  appropriate  clinical  setting. 

Vendor  relationships.  Establish  links  with  relevant  equipment  vendors  and 
other  product  vendors  for  clinical  demonstration  support,  participation,  and 
licensing  agreements. 

FDA  approval.  Pursue  course  of  action  required  for  FDA  approval  of  system 
software,  as  appropriate. 


Conclusions 

The  value  of  this  interface  integration  approach  to  clinicians  has  been 
demonstrated.  It  has  also  become  clear,  however,  that  imagining  effective 
applications  of  novel  inclusive  interfaces  can  be  very  difficult  for  many  of  us. 
This  is  especially  challenging  for  multi-participant  scenarios,  as  we  try  to 
imagine  the  interaction  effects  among  individual  data  representations,  or 
methods  for  jointly  configuring  shared  display  real  estate  (a  wall  panel 
display,  for  instance).  Design  of  these  multi-modal  "egocentric"  data 
interfaces  is  an  area  ripe  for  visualization  using  VR  simulation  of  a  more 
dynamic  and  realistic  clinical  environment. 

The  LIMIT  testbed  developed  under  this  DARPA  grant  now  provides  a 
resource  for  virtual  prototyping  and  evaluation  of  a  host  of  new  medical 
devices  and  patient  data  representations.  As  products  begin  to  emerge  from 
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the  Advanced  Biomedical  Technology  program  we  will  be  able  to  more 
rapidly  integrate  them  into  a  clinically  acceptable  user  interface. 
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Virtual  Emergency  Room 


To  create  the  Virtual  ER,  HITLab  artist  Peter  Oppenheimer  mapped  photographs  of  the 
Harborview  Medical  Center’s  trauma  unit  as  a  continuous  texture  map  onto  a  cylindrical 
geometry.  Such  an  approach  provides  a  strong  sense  of  presence  within  the  environment 
and  is  suitable  for  situations  in  which  the  user’s  virtual  position  remains  relatively  stable. 
Image  data  courtesy  of  Don  Stredney,  Ohio  Supercomputer  Ctr.;  the  NLM  Visible  Human 
project;  the  EMBBS  Web  site;  J.W.  Rohen  and  Yokochi’s  Color  Atlas  of  Anatomy;  Peter 
Dunbar,  MD;  and  Harborview  Medical  Center,  Seattle. 


It  might  not  be  the  stuff  of  a  hit  TV  show, 
but  a  recently  built  ER  is  a  ‘HIT'’  product 
nonetheless.  As  part  of  an  ARPA  research 
grant  to  study  advanced  medical  re¬ 
sources,  the  University  of  Washington's 
Human  Interface  Technology  Lab  (HIT¬ 
Lab)  has  developed  a  virtual  emergency 
room  in  which  medical  personnel  can 
explore  the  design  space  for  medical  in¬ 
terfaces  of  the  future.  The  environment 
will  be  used  to  help  physicians  and  plan¬ 
ners  determine  how  immersive  augment¬ 
ed  space  might  best  be  utilized. 

Inside  the  virtual  ER.  users  can  grab 
various  items,  such  as  X-ravs,  EKG  read¬ 
ings,  and  heart  rhythm  strips,  and  place 
them  anvwhere  in  the  space.  Each  object 
can  be  stabilized  relative  to  a  number  of 
different  reference  points,  such  as  the 
room,  the  patient,  and  the  user’s  head 
and  body,  and  objects  can  be  made  visi¬ 
ble  or  invisible  at  the  user’s  command. 

The  HITLab's  virtual  ER  is  a  replica  of 
the  trauma  center  at  Seattle-based  Har¬ 
borview  Medical  Center.  According  to  re¬ 
searcher  Suzanne  Weghorst,  director  of 


interface  design  at  the  HITLab,  the  envi¬ 
ronment  was  created  using  a  technique 
similar  to  QuickTime  VR.  "Frames  from  a 
video  capture  of  the  Harborview  trauma 
unit  were  stitched  together  and  massaged 
in  Photoshop  on  a  Mac  into  a  seamless 
texture  map,  which  we  projected  onto  a 
cylindrical  ’room’  in  the  VR  system,  along 
with  texture-mapped  virtual  data  objects 
[created  with  software  from  Alias/Wave¬ 
front  and  a  patient  model].” 

The  environment  is  currently  running 
on  an  SGI  Onyx  RealityEngine2  using  VR 
software  from  Division  and  Senses.  A  Pol- 
hemus  motion-tracking  system  captures 
head  and  hand  motion,  and  an  in-house 
wand  facilitates  navigation.  Various  PC- 
based  voice  recognition  packages  are 
being  investigated  to  incorporate  voice 
commands,  but,  according  to  Weghorst, 
"this  has  given  us  lots  of  problems  in  our 
noisy  lab  environment.” 

As  with  all  ambitious  VR  projects,  the 
success  of  the  virtual  ER  depends  on  the 
degree  to  which  the  feel  of  the  space  is 
believable.  According  to  Weghorst,  “The 


TECH 


WATCH! 


texture-mapped  room  works  well  for 
creating  a  strong  sense  of  presence  in  a 
clinical  environment,  but  now  we  are 
challenged  to  incorporate  real-time  pa¬ 
tient  data  streams  and  interaction 
metaphors  and  techniques  that  are  ap¬ 
propriate  for  each  data  object."  In  addi¬ 
tion,  she  notes,  "this  application  cries 
out  for  multiple  live  video  textures,  so 
we  can  show  more  realistic  dynamic  ob¬ 
jects,  such  as  ultrasound  studies  and 
teleconsultation.” 

The  HITLab  has  submitted  a  proposal 
to  ARPA  for  the  next  two  phases  of  the  Ad¬ 
vanced  Medical  Interfaces  project.  The 
second  phase  involves  building  a  more 
robust  and  realistic  simulation  environ¬ 
ment  that  will  enable  the  researchers  to 
simulate  clinically  meaningful  tasks  and 
to  perform  usability  assessment  and  sys¬ 
tem  specification  studies.  The  third  phase 
involves  the  design  and  implementation 
of  a  spatial  data  display  system  that  will 
incorporate  an  array  of  medical  interface 
technologies  currently  under  develop¬ 
ment.  The  system  will  be  tested  in  an  ac¬ 
tual  clinical  setting. 

As  the  project  progresses,  says 
Weghorst,  “our  biggest  challenge  is  com¬ 
ing  up  with  visionary  inspirations  that  are 
both  technically  plausible  and  clinically 
useful.  It  requires  constant  iteration  and 
cross-checking  between  our  engineers 
and  our  consulting  physicians.”  • 
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